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Asymptotic Joint Distribution of Extreme 
Eigenvalues of the Sample Covariance 
Matrix in the Spiked Population Model 



Dai Shi^ 

O 

Abstract 

. In this paper, we consider a data matrix X G £^NxM ^j^g^g ^j^g 

I columns are i.i.d. samples being dimensional complex Gaussian of mean 

zero and covariance S G C^^^. Here the population matrix S is of finite rank 
perturbation of the identity matrix. This is the "spiked population model" 
i first proposed by Johnstone in [18]. As M, N ^ oo but M/N = 7 e (1, 00), 

Ph ■ we first establish in this paper the asymptotic distribution of the smallest 

• eigenvalue of the sample covariance matrix S := XX* /M. It also exhibits 

"j^ ! a phase transition phenomenon proposed in |lj — the local fluctuation will 

be the generalized Tracy- Widom or the generalized Gaussian to be defined in 
the paper. Moreover we prove that the largest and the smallest eigenvalue 
are asymptotically independent as M,N — )• 00. 
J> I Key Words: Spiked population model, asymptotic independence of the ex- 

■ treme eigenvalues 

O 

^ ■ Suppose we have M independently and identically distributed samples xi, . . . , xm G C^, 

each of which is an dimensional complex- valued vector. Here M is the sample size and 
N is the dimension of our data. We can then form the data matrix X = {xi, . . . , xm) ^ 
(^NxM further define its sample covariance matrix 



1 Introduction 



S := —XX* G C^^^. 
M 

In this paper we are interested in the asymptotic joint distribution of the largest and the 
smallest eigenvalues of the matrix S. Below is the assumptions of our model. 

• All the data vectors Xi are independently and identically distributed following the 
complex Gaussian distribution of mean zero and covariance matrix S G C^^^. 
Here S is a non-random positive definite matrix. 

• M, X — 7- 00 but their ratio M/N := 7^ is a fixed amount in the interval (1, 00). 
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• We denote ii > £2 > • • • > to be the eigenvalues of the matrix S. Then we 
assume that all of the ^j's are equal to one except for only finite of them. That is, 
there exist fixed integers ri,r2 which are independent of M,N such that 

£n < £n-1 < . . .< ^7V-ri+l < 1 < ^2 < 4.2-1 < • • • < ^1 

and 

^N-ri = ^N-ri-l = ■ ■ ■ = ir2+2 = ^2+1 = 1- 

The model defined above is the (complex) "spiked population model" proposed in 
|18] . The unit eigenvalues represent pure noise, while the spiked eigenvalues represent 
true information. In real applications, we will encounter such models quite often. In 
mathematical imaging (see [21]), the observed spectrum of the sample covariance matrix 
indeed has some detached eigenvalues, representing the possible scatterers in the region. 
As another example, in mathematical finance (see [22]), each column of our data matrix 
represents the correlated return of each stock. The sample correlation matrix has some 
spiked large eigenvalues, representing the main factors driving the market, and some 
small eigenvalues, representing the linear dependence of these factors. Other possible 
applications include, but not restricted to, speech recognition (see [23]), physics mixture 
(see [23]) and statistical learning (see [2S])- 

We define (Ai, . . . , Aat) as the eigenvalues of the sample covariance matrix 5", where 
Ai > A2 > . . . > Aat. In the null case where S = /, a lot of properties are known. 
The empirical measure of {Ai}^^, denoted by Fn ■= ^^li '5(Ai)/A^, will almost surely 
converge in distribution to the Marcenko-Pastur law (see [ID]), whose density is defined 
by 

F(A) := ^v/(A+-A)(A-A_)- 1{A_<A<A,} (1.1) 

where A+ := (1 +7^"*^)^ and A_ := (1 — 7^"*^)^. The support of the density, [A„,A+], is 
often called the Marcenko-Pastur sea. Regarding the largest eigenvalue Amax = Ai and 
the smallest eigenvalue Amin = Atv of the sample covariance matrix S, German first proved 
that Amax — ^ A+ almost surely in [11] and later Silvertein proved that Amin A_ almost 
surely in \L2i. That is to say, the largest and the smallest eigenvalues will converge to 
the corresponding edges of the Marcenko-Pastur law. For a second order approximation, 
Johansson in [TJ] proved that Amax, properly scaled and centered, will converge weakly to 
the Tracy- Widom law. Baker, Forrester and Pearce in [T7] also proved the similar result 
for the smallest eigenvalue. More precisely, we have 



P 
P 



(Amax-(l+7-Y)-yY^<a: 
(1 - 7"')' - Amin) 



(1 + 7)'/' 

7M^/3 

(7 - 1)4/3 



^ Fo(x), (1.2) 
^Foix). (1.3) 



Here -fo(") is the cdf of the Tracy- Widom law, i.e. Fq{x) := det(/ — Ax) where Ax is 
the Airy operator on L^(x,oo) with kernel defined later in (11.81) . Later we will explain 
why -Fo(-) has this zero subscript. We note that most of these results are universal, as is 
proved in [H], [19] and [20], to list a few. 
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For the spiked population model where S 7^ /, the phenomenon becomes much 
more interesting. Recent research found that the non-null eigenvalues tend to pull the 
extreme sample eigenvalues out of the Marcenko-Pastur sea [A_, A+], provided that they 
are larger or smaller than certain thresholds. In [3] Baik and Silverstein proved the almost 
sure limits of the extreme sample eigenvalues pulled out by the spikes. He proved that 
for fixed j, if the largest j-th population eigenvalue ij is greater than 1 +7"^, then the 
largest j-th sample eigenvalue Xj satisfies 

^ + 'T — 7 almost surely. (1.4) 

ij — I 

If ij is less than or equal to the threshold 1 + 7"^, then Xj — )■ A+ which is just the 
right edge of the Marcenko-Pastur sea. The same is true for the smallest eigenvalues. If 
iN-j+i < 1— then fll.4p also holds for j replaced by N—j + 1. Otherwise X^-j+i — ?■ A__ 
almost surely. Note that this includes the case where some of the £j's are the same. In 
this case, the corresponding Aj's just converge to the same limit specified in (11. 4p . We 
call these eigenvalues "packed". 

But what is the second order approximation? Baik, Ben Arous and Peche in [1] 
observed the phase transition phenomenon of the asymptotic distribution of the largest 
sample eigenvalue A^ax = Ai. They proved that if £1 > 1 -|- 7"^^ (i.e., when Ai is pulled 
out of the sea), then the local fluctuation of Ai will be asymptotically the same as the 
largest eigenvalue of a x A; GUE matrix, where k is algebraic multiplicity of ii. On the 
other hand if £1 < 1 + 7"^ then Ai follows the generalized Tracy- Widom law defined in 
Definition 11.11 Moreover, in |3] Bai and Yao obtained the joint local fluctuation of the 
packed sample eigenvalues which are pulled out — when suitably centered and scaled, 
they are asymptotically the same distributed as some unitary Gaussian random matrix. 

We note that similar results can also be obtained for perturbed GUE case, see [2] 
and [6J. In [1], [5] and [7] the authors also got the similar results for real Gaussian case 
or even universal case. 

However, none of these results deal with joint distribution of the largest and the 
smallest sample eigenvalues in our spiked population model. Intuitively they should 
be independent. Indeed, the distance between the two extreme eigenvalues is at least 
A+ — A_, i.e., the width of the Marcenko-Pastur sea. For this large distance, the repulsion 
force between eigenvalues is very weak. As a result, asymptotically they will fluctuate 
freely, without interacting each other at all. For a brief history, in Bianchi, Debbah 
and Najim first established the independence for the GUE case. Their method is based 
on bounding the expansion of the Fredholm determinant. Later Bornermann in [9] used 
operator theory to give a much simpler proof. In [16] a different approach relying on 
PDE can also be found. 

In this paper, the two branches of the results are combined. Here we establish the 
asymptotic distribution of (Ai,A7v), i.e., the largest and the smallest eigenvalues. More 
precisely, we have two results. The first one is for the asymptotic distribution of the 
smallest eigenvalue. Here we also observe the same phase transition phenomenon as in 
[1]. We proved that, when suitably centered and scaled, the local fluctuation of the small- 
est eigenvalue will be similar to the largest one discussed above. As a second result, we 
prove that as M, — > 00, the largest and the smallest eigenvalues will be asymptoti- 
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cally independent. From this we can easily establish the asymptotic joint distribution of 
(Ai, X]\f). 



1.1 Main Results 

To state our maim theorems, we need to define a few functions. They first appeared in 
pp. In that paper Baik, Ben Arous and Peche also discussed some properties of them. 
For any integer m > 1, we define 

= e-^-'/'^da, (1.5) 

t^"'\v) = — [ e™"+"'/3(-m)'"-^c/a. (1.6) 

271 J 

Here the contour is from ooe^'^^^ to ooe*'^/^ such that the point a = lies above the 
contour. As an immediate observation, s^^^ = t^^^ is just the Airy function, denoted by 

Ai(M) := — f e''"'+'°'''^da (1.7) 
2vr J 

with the same contour as above. We can also define the Airy kernel 

, Ai(M)Ai'(t;) - Ai'(u)Ai(w) p^., 

A(M,f) := — — — — — —= I ki{u + y)ki{v + y)dy. (1.8) 

u-v Jq 

Furthermore, we define Ax as the integral operator on LF'{x, oo) with the kernel A(M,f). 
Definition 1.1 Given an integer k > 0, Fk{x) is defined to be 

Fk{x) = det(l - A) • det(Smn - (T^^^^™^ (1-9) 

V \ -^x I J rn,n=l 

where (■, ■) is the inner product in L^{x, oo). 

Remark 1.1 As is shown in fl]/, -Ffc(-) is a distribution function. Moreover, Fo(-) is 
just the basic Tracy-Widom law. Hence Fk{-) for general k > can be regarded as a 
generalization of the Tracy- Widom law. 

Definition 1.2 For k > 1, we define the distribution function Gk{x) as 

k 



GJx) := 



to be the distribution function of the largest eigenvalue of a standard k x k GUE matrix. 

Remark 1.2 When k = 1, Gi{-) is just the density of a standard Gaussian. Hence we 
can regard Gfc(-) as a generalization of the Gaussian distribution. 
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Having defined our key distribution functions Ffe(-) and Gfe(-), we can state our main 
theorems below. 



Theorem 1.1 For the spiked population model defined above, the smallest eigenvalue 
Amin = of the Sample covariance matrix has the following asymptotic distribution, 
depending on whether the smallest eigenvalue of the true covariance matrix E is small 
enough or not. 

1. If En = ■ ■ ■ = (^N-k+i = 1 — for some k >0, and the other £i 's lie in a compact 
subset of (1 — 7~^, oo), then as M — >■ oo 



P 



7 



MV3 



(7 - 1)4/3 



<x}^ Fk{x). 



2. If En = ■ ■ ■ = (^N-k+i < 1 — 7 ^ for some k > 1, and the other ii 's lie in a compact 
subset of {iisi, oo), then as M ^ oo 



P 



.1/ 



1/2 



£n1 



-2 



'N 



'^mir 



<x}^ Gk{x). 



;i.i2) 



Our next tlieorem states tliat as M ^ oo, tlic largest and tlie smallest eigenvalues 
of the sample covariance matrix are asymptotically independent of each other. 

To state it more rigorously, we assume that — . . . — ii — . . . — for 

some ki,k2 > 1, and the rest of the iiS lie in a compact subset of {InAi)- We define 
/ci, k2 such that 



ki 



if^Ar>(l-7-')', 
fci if£Ar<(l-7-')'- 



k. 



if ^1 < (1 + 7-1)2, 
k2 if^i > (1 + 7-')'- 



(1.13) 



For (Amin, Amax) = (Aat, Ai), wc define the centered and the scaled version of them. Define 



A. 



A. 



(7-1)4/3 



(1-7-T-A. 

Ml/2 



■N 



iNl- 



Amir 



if^iv>(l-7~')' 

1,2 (1-14) 



if 



< (1-7" 



(7+1)4/3 



A. 



Ml/2 



V^?+^?7-V{^i-iP 



(1 + 7-T 



if 4 < (1 + 7"')' 
if £i > (1 + 7"')' 



;i.l5) 



to be the centered and the scaled version of (Amin, Amax)- Then we have the following 
theorem regarding the asymptotic distribution of (Amin, Amax)- 

Theorem 1.2 Under the assumption of the spiked population model, as M ^ oo 



Fu,{x)FkM iflN>{l-l 



-1\2 



<(l+7- 



-1\2 



^(Amin < X, Amax < V) ^ 



GkAx)FkM tfiN < (1 - 7-')',^i < (1 + 7"')', 
Fk,{x)GkM if In > (1 - 7-')',^i > (1 + 7-')', 
GMGkM if^N < (1 - 7-')',^i > (1 + l-'Y- 
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Remark 1.3 Since -Ffc(-) orGk{-) is the asymptotic distribution o/Amin and Amax, The- 
orem \l.^ also tells us that (Amm, Amax) are asymptotically independent. 

Possible applications of our result include data analysis. Suppose we have a large 
data matrix X G C^^*^. The question is to determine whether the samples are i.i.d. 
drawn from a Gaussian distribution of a given spiked covariance matrix S. For example, 
when E = / we just want to determine whether our data matrix is pure noise. One 
possible way is test the hypothesis is to see whether the largest and smallest eigenvalues 
are close to their asymptotic limit. But here we are testing the null hypothesis using two 
criteria, involving both A^ax and Amm- Hence we need their joint distribution to carry 
on the hypothesis testing. Another possible application involves the local fluctuation of 
condition numbers. They are both discussed in Section [2j 

The rest of the paper is organized as follows. A sketch of the proof will be provided 
in section [Lll In section [2] we will discuss some applications of the result. In section [3] 
we will represent the probability P(Amin < x, Amax < y) in a determinantal form, which 
will be the basis for future proofs. Section H] and E] will be devoted to prove Theorem 11.11 
under the two cases. The proof of Theorem 11.21 can be found in section [61 Finally Section 
[7] serves the conclusion of the whole paper. 

1.2 Sketch of the proof 

First let's focus on the smallest eigenvalue. The basic idea is to calculate the large M limit 
of the density of Amin- Since in our model, the distribution of each sample is Gaussian, 
we can write down the density explicitly in (13.11) . Due to some trick proposed first by 
Tracy and Widom in [12], we can further write the density into the form of a Fredholm 
determinant det(/ — /Cn), where the kernel of the operator /Cn is defined in (13. lip . We 
note that the kernel can be written as two contour integrals. By designing our contours 
carefully and by the saddle point analysis, we can finally obtain the asymptotic limit 
of the kernels, or the limit of the operator /Cn itself under the trace norm. Since the 
determinant is a locally Lipschitz function of the operator under the trace norm, we have 
our desired limit. 

For proving the asymptotic independence, our strategy is the same. We write the 
joint distribution of (Amm, Amax) into a determinantal form det(/ — /C) in (13. 2p . where the 
operator K, = {K,ij)fj^i can be written as a 2 x 2 block matrix. We prove that under the 
correct scaling, the diagonal blocks /Cii,/C22 will have a non-trivial limit, while the off- 
diagonal blocks will converge to zero, both under the trace norm. Hence K, will converge 
to a block diagonal matrix. The independence result follows easily from the trivial fact 
that the determinant of a diagonal matrix equals to the product of the determinants of 
its diagonal blocks. 

2 Applications 

In this section we will discuss some applications of Theorem 11.21 In subsection 12.11 we 
will talk about the asymptotic distribution of condition numbers of a data matrix. In 
subsection 12.21 some application in hypothesis testing will be discussed. 
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2.1 Distribution of Condition Numbers 



As a simple consequence of Theorem 11.2^ we can get the asymptotic fluctuation of the 
ratio Amax/ Amin, whlch is the square of the condition number of the original data matrix 
X. More precisely, we have the following Corollary. 

Corollary 2.1 Let k, be the condition number of the original data matrix X . With the 
definition of ki,k2 in U.13\} . we have the following four results. 



1- If^N>{l-T^f,^i<{l + Ty, then 



1-7' 
1 + 



'In 2 



7 



7 



1)2/3 ^^^fel ' + l)2/3'^^'=: 



(2.1) 



2. If In < {l-T'^fAi < {l + T'^f, then 

'^N-^NTV{'^-^N) 



Ml/2 



:i+7- 



-1\2 



v/l-[7(l-^^) 



1- 7-2(1 



(2.2) 



3. If In > (1 -7"i)2,£i > (1+7-1)2, then 

(1-7-T 



Ml/2 



h + iii-Viii-i) 

I If In < > (1 + 7-')', then 

Ml/2 



y/l - [7(£l - l)]-2 

l + 7-2(£i-l)-i 



(2.3) 



^jv-^iv7-7(l-^Jv) 2 ; 

■K — 1 



^1 + ^17-7(^1-1) 



— 



v/l-[7(l-M- 



1-7 



-2/ 



r^G,, + — 



Vi - [7(^1 - 1)]" 



+ 7-^(^1-1) 



-1 



(2.4) 



Here X, Y are two independent random variables with distribution specified in their sub- 
scripts. The convergences in \2.1\ -\2.4\) all mean convergence in distribution. 



Proof. Here we just prove the corollary for case 1. By Amin, Amax in f ll.l4p and f ll.lSp we 
have 



M2/3 



1 -7-i\2A 



1 + 7-1^ Ami 



1 



(1-7-^) 



7 



(7 - 1)2/3 



Amin + 



7 



(7 + 1)2/3 



A. 



(2.5) 



In (12.51) the first factor will converge to one in probability. By Theorem 11.21 the random 
variable in the bracket will convergence to the right hand side of (12. ip . By Slutsky's 
lemma our proof is complete. The proofs for the other three cases are the same. □ 
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2.2 Hypothesis Testing 



In this section we discuss one way to test whether the columns of the data matrix X are 
i.i.d. samples from complex normal distribution of mean zero and covariance matrix S. 
This is our null hypothesis to be tested. Note that sometimes our goal is just to test 
whether our data is just pure noise, where we have E = /. But the method discussed 
in this subsection can be generalized to any other covariance matrix E as long as it is of 
finite rank perturbation of the identity. 

To save space, here we just discuss the case where ^at > (1 — < (1 + 7^^)^- 

The method for other cases can also be very similarly obtained. By Theorem II. 2[ we 
know that Amax, Amin will converge to (1 + 7~^)^ and (1 — 7~^)^, respectively. Intuitively, 
if the observed two extreme eigenvalues of the sample covariance matrix deviates from 
the two limits by a large amount, then probably we should reject the null hypothesis. 
However, here we are testing the null hypothesis using two criteria. We claim that Amin 
should be close to (1 — 7"^)^ and Amax should be close to (1 + 7^^)^. Thus we need to 
calculate the joint distribution of the two. Hence our Theorem 11.21 plays an important 
role here — it claims that they are asymptotically independent. 

Here's our method to test the null hypothesis, step by step. 

1. Form the sample covariance matrix S = XX* /M from the data matrix X. Then cal- 
culate the observed maximum and minimum eigenvalue of it, denoted by (Amax, Amin)- 

2. Use f ll.l4p and fll.lSp to calculate the centered and scaled version of (Amax, Amin), 
denoted by (Amax, Amin)- 

3. Calculate our statistic T := (1 — -Ffe^(Amin))(l — Fk^iXmax))- If T < a where a 
is a pre-determined confidence level, then we reject the hypothesis. Otherwise we 
accept it. 



3 Determinantal Form of the Distribution 

In this section we will derive the determinantal form of the distribution of the extreme 
eigenvalues. This formula will be the basis for future saddle point analysis from Section 
m to ini From now on, we will always denote C, c as some positive constants independent 
of M, A^. Their value may vary from line to line, but they are always constants. 

For notational convenience, we define vTj = for i = 1, . . . , N. Then if vr^'s are 
distinct, the joint probability density of (Ai, . . . , X^) can be written as the following form 

p{Xu...,Xn) = ^det(exp(-M7r,A,))J^_^-^n^''"''- (^.l) 

for some constant C > 0. Here V{x) := nj<i("^« ~ -^i) Vandermonde determinant. 

If some of the vTj's coincide, we interpret (13. ip using the L'Hopital's rule. 

For any two real numbers < .^1 < ^2 < cxd, let Ji, J2 be two disjoint intervals on M 
such that 

Jl = (0,ei), J2 = (6,00). 
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Moreover, we define J := Ji U J2 and we denote E{J, 0) as the probability that there are 
no eigenvalues in the set J . That is, 

E( J, 0) = P(ei < A^in < A^ax < 6)- 

Our first proposition gives a determinantal representation of E{J,0). This determinant 
form will turn out to be an invaluable formula for future analysis. 



Proposition 3.1 We have 



E{J,0) = det J 



Kn K 



12 



(3.2) 



K21 K22 

where for a, /3 G {1, 2} Kjj^a is the integral operator on Jq,) — > L'^lJjs) with the kernel 

K,AvX) = -7^2[ dz [ d^e-''^(^-^^)+^^(--^^)^n^^^-f-r- (3-3) 
(27r)2 w-z z - tiu \w J 



Here qi,q2 are two pre- fixed constants such that 



(3.4) 



< g2 < niin{7rj}^-^i < max{TTj}^^^ < qi. 

Moreover r^(/3 = 1,2) is the contour on C enclosing {vTj}^]^ and lying in the strip 

Ti3C{zeC:q2< ^{z) < q,}. 

Hi = {z = A + it : t & M} is a vertical line from bottom to top for A > qi. II2 is counter 
clockwise circle enclosing the origin and lies in S2 C {2 : 3f?(z) < qi}- These contours 
are sketched in FigureUl 




Figure 1: Graph for the contours Si, S2 and T 



Remark 3.1 Here qi,q2 can be any constants as long as is satisfied. They will 

not affect the determinant in i\3.2\) . 
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Proof. The proof is almost the same to the one in [Tj. Thus we just point out our major 
differences. The rest will be quite sketchy. Since we have an explicit expression for the 
probability density fl3.ip . we can express E{J,0) into the following (for details, see (66) 

of H). 

E{J,0) = - ... det(Ar^)det(e-^^-^^'=)n(l-Xj(A.))Af-^rfA. 

Jo Jo 

= ^det(^°'(l-x.(A))A^-i+^^-^e-^^-^rfA)''^ ^ 

= IdetfA,,- / \^-'+^'-^e-^'^'^d\Y . 
^ \ J J J j,k=i 

Here Xj(-) is the indicator function of the set J and Ajk is defined to be 

POO 



If we define A = (^jA:)j^fc=i) then we can show that 



dct.i-fT ^^^'+^~^^ n 



^<j<k<N 



Now we define (pf\X) = \J'^+M-N^-Mg,\ (l>^\\) = g-^^^^^-^^)^ for i = 1,2. Here 
qi,q2 are two pre- fixed constant such that condition f l3.4p is satisfied. Then we observe 
(j)f{X), $f (A) G L^{J). We also define the operators F : L'^{J) ^ M^, G : ^ L'^{J) 
by 

F:/(A) ^ ([ 4>f\x)f{X)dX+ [ <pf\X)f{X)dxY 

\Jjl Jj2 / j = l 

N N 

^ J] )(A)xj,(A) + a,$f (A)xj,(A). 
i=i i=i 

Then we can re- write E{J,0) as 

E{J,0) = ±-det{A~FG) = =^det{I-A-^FG) 

= C'det{I -GA-^F). (3.5) 

Please note that GA^^F is an operator on L'^{J) — ?■ L'^{J) and G' is a constant inde- 
pendent of J. We note that GA~^F = PjGA~^FPj can be also regarded as an operator 
on L2(0, oo). Recall that J = (0,6) U (6, oo). With and ^ ^ oo, G^-^F will 

converge to zero. Hence take J — )■ in (13. 5p gives C" = 1. 

Now we note that L^(J) = L^(Ji U J2) is isomorphic to L'^{Ji) © L^(J2) under the 
trivial isomorphism 

/ G L\J, U J2) ^ (/I j„ /IjJ G L2(J0 © L2(J2). 
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Thus, under the isomorphism, (13 .Sp can be further written as 



Here Pj^^GA^^FPj^ is an operator on L^{Ja) — ^ L'^{Jp). Following the rest of the proof 
of Proposition 2.1 in pQ, we can obtain that, for G {1,2}, 



{PjfiA-'FPj:){r,X) 



(27r)2 Jp^ J^^ w-z Al z-Tii \w 



■ (3.7) 



Here F/j and Sq, are the contours defined in Proposition 13.11 The only difference is that 
in our case, instead of using equation (76) in P, we used 



for both 13 = 1,2, as long as M — = (7 — 1)A^ is large enough. □ 

If we consider the special case of Proposition 13. II when Ji = (0,^) and J2 = 0, then 
we arrive at the following Corollary. 

Corollary 3.1 We have 

nXmin>0 = detiI-Kn) (3.8) 
where Ku is the integral operator on L'^{0,^) — ?■ L^(0,^) with the kernel defined in 113. 3\} . 

As the first goal of this paper, we will focus on the distribution of the smallest 
eigenvalue Amin- F)ue to Corollary 13. 1[ we can in turn analyze the asymptotic behavior 
of the operator Ku. 

First we note that the kernel function Kii{r], Q can be simplified. Since the contour 
F is always on the left of Si, then "iz G Fi, w G Si we have '^{z — w) < 0. Hence we have 

-^ = Mj^ exp!^-yM{w-qi-iz-qi))'^dy. (3.9) 

Substitute (13. 9 p into (13. 3p . we obtain 



Ku{vX) = - H,{y-r,).h{y-C)dy (3.10) 
^0 

where 

M /■ r 1^1 

^i(^) = TT exp r/M(z-gi) -z^'^n (^-H) 

/[ r ( ^ ^ 

- expl -CM{w - qi) yuj-^^ JJ(w - ne)duj. (3.12) 
^ J Si L ) i=i 



MO = 2 
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Now we use the change of variables. Define 

^ = f^i-^x, T] = fii- ^{x + u), ( = fi^-^{x + v) (3.13) 
where /ii, z/i and a are constants that will be defined later. Under such change of variables 



IP(Amin > = det(J - Ku\{o,o) = ^^^(^ " ^iilccm-'^i^/Af-)) = det(/ - Ki 
where 

/Cii(M,f) = -T^Kiil 111 ~ —^{x + u), Hi TT-fx + f' 



l\{0,^ilM°'/ul-x)J 



poo 

/ ni{y + x + u)Ji{y + x + v)dy. (3.14) 
Jo 



Here 

n-a - ^ 



Uiiu) = "^^LJl f e-^^-^^'^^-'i)+'^''^'-"<^~'i-) -z^'f] (3.15) 

Under the assumption that only ri + r2 of these vr^'s are not one, we can further write 
'Hi{u),Ji{v) as 

Ui{u) = / g.M-<^.(.-.,)+M/.W . TT ^^dz, (3.17) 

"^^ £<r2 or ^>Af-ri+l 

Ji(^) = / ^-uM^-vi.^,.)-MMu) . TT ^^i^rf,^. (3.18) 

"^^1 £<r2 or £>Af-ri+l 



Here 



fi{z) := -/ii(z - qi) + \ogz- \og{z - V)lf (3.19) 



where the log function is defined in the principal branch. According to the discussion in 
P, we just need to find suitable limiting functions 'Hi,oo(m), J7i,oo('y) G L^(0,oo) and a 
constant Zm such that, for any x in a compact set, 

/»00 /"OO 

/ / \ZMUi{x^u^y) -Ui^^{x^u^y)'^dudy -f^, (3.20) 

/•OO /'OO 

/ / \Z~^Ji{x^u^y)-Ji^^[x^u^y)fdudy^^. (3.21) 
Jo Jo 

In the following two sections we will use the saddle point analysis on /i(-2) to realize our 
promise. 
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4 Proof of Part 1 of Theorem 11.1 



In this section we will prove Theorem 11.11 for the case ^at > 1 — 7^^- Since in this 
section and the next section we will focus on only /Cn defined above, thus we temporarily 
drop all the subscript one, for notational simplicity. For example, instead of writing 
/ii, z/i, Fi, "Hi (u), 7^1,00(^)5 we will write /i, F, f{z),'H{u),'Hoc{u). Note that these 
are only valid in this and the next section. 

We assume that for some < < ri (recall tt^^ = ii), 



= = ■■■ = = 1-7 



and the rest of the vr^'^'s are in a compact subset of (1 — 7"^, 00). The rest of the job is 
to provide a saddle point analysis of T-L^u) and J7'(f). 
In this case, we can define our constants to be 

Here e > is a pre-fixed smaller number and p will be later shown to be the saddle point 
of f{z). We also define 

= _ iL..-. n (^--^)- (4-2) 

^ ^ e<r2 or N~ri+l<i<N-k 



Then our Ti^u), J{v) can be written as 



U{u) = / e-"^''-"-^^-"^ ■ e''f^-^^ ■ -j-^^^—-dz, (4.3) 



Our constant Zm in this case is defined to be 

7 ^ 9ij>) .^ 

Finally, as we promised in f l3.17p and fl3.18p . our Tiooiu) and Jooiy) are defined as 

r -1 1 

= -IT- / (4-6) 

^oo(t;) = 1^ / e-'^'^'^'/'b'db. (4.7) 

Here Fqo is the contour from ooe~^'^/^ to ooe^'^/'^ such that a = lies on the left hand 
side of the contour. Eqo is the contour from ooe^'^^^ to ooe'^^^. The main target of this 
section is to prove the following Proposition. 

Proposition 4.1 Assume that u and v are hounded below. That is, assume that there 
exists some U^V eM. such that u > U,v > V . Then 
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1. There exists some constant C, c > 0, Mq > such that uniformly for u > U , 

\ZMn{u) - Hoo{u)\ < Ce-'^'^M-^/^ yM>Mo. (4.8) 

2. There exists some constant C,c> 0, Mq > such that uniformly for v > V, 

\Z-,lJ{v) - Jo.{v)\ < Ce-^"M-l/^ VM > Mq. (4.9) 

In order to prove Proposition 14.11 we perform the saddle point analysis on f{z) 
defined in (13.191) by f{z) := —fJ,{z — q) + log z — 7"^ iog{z — 1). Through some very simple 
calculation, we find that 

f'ip) = f'ip) = 0, f"'{p) = -2v\ (4.10) 

Hence p is the saddle point of f{z). Since f"'{p) is negative, the steepest descent direction 
for e^-^'-^-' appearing in l-Liu) is with angle ±27r/3 to the real axis. The steepest descent 
direction for e~^'^^^^'' in J{v) is with angle ±7r/3 to the real axis. Let's analyze 'H{u) 
first. 

4.1 Analysis of 

To carry out an saddle point analysis, we need to define our contour F for 'H(-u) first. In 
the following we only define F in the upper half of the complex plane. The part in the 

lower half is just of a reflection with respect to the real axis. Let F := Ui=o Ui=o 
where 

where e is the small constant appeared in the definition for q in (14.11) . Define F2 to be 
the arc centered at the point z = 1 and connects the endpoint of Fi. We define 

p* = (4.11) 

^ 1 + 7 

For the definition of the rest of the contours, we need to split it into two cases. 

• Case 1. If F2 does not intersect the vertical line {z = p* + it : t E M.}, then F4 is 
defined to be a horizontal line just above the real axis, i.e. 

r4 C {2 = t + m : t G M+} (4.12) 

for some small fixed constant k > 0. We also require that F4 should connect the 
endpoint of F2 and the point min{7ri,p*} + in. F5 is a quarter circle centering at 
min{7ri,p*} + ifi;, with radius k, just connecting F4. For a graph of the contour, 
please refer to the left part of Figure El 
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Figure 2: The Contours T. Case 1: Left. Case 2: Right. 



Case 2. If r2 intersects the vertical hne {z = p* +it : t & M}, then we just define 
to be the part of that hne, connecting the endpoint of r2 and heading downward. 
We add Fa into our contour. The definition for F4 and F5 are exactly the same as 
that in Case 1. For a graph for this case, please refer to the right part of Figure [2l 



We choose the contour F = IJLo ^ Ui=i ^Vich a specific way to establish the 
following lemma. 

Lemma 4.1 In both cases, ^{f{z)) is monotonously decreasing when z travels along 
the path |J^=i Tt, provided that we choose k small enough. For F4, F5 we have 

sup ^{f{z))<^{f{z*)) 

where z* is the intersection o/Fi and F2. 
Proof. We divide the proof into several parts. 

• In Fi, z = p + te^^''^ for {2pM^/^)-^e < t < (Vs - - 1). Then 

^ {l-l)H^r + {2p-l)t-2p{p-l)] 

dt '' 272 [t2 + (1 _ p)t + (1 - p)2][t2 _ pi + p2] • 

Everything is positive except the quadratic term t^ + (2p — l)t — 2p{p — 1). When t 
is taken to be its largest possible value (v^ — 1), the quadratic term achieves 

it maximum, which is -73(^3- 1) < 0. Hence we get d^{f{z))/dt < 0. 

• In F2 we have z = 1 + i?e^*^ for 9 taking value in a subset of (0, vr). The R is some 
constant. We have 



d_ 

de 



< 



1 + R'^ + 2Rcose 

Rsm9 
1 + R'^ + 2Rcos9 



+ R^ + 2Rcos( 



fi{R+l) 



(4.13) 
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But we can calculate R = (p — 1) ■ (3 — a/3)/-\/2 < p — 1 = — 1. Hence 
d^{f{z))/d9 < 0. 

In Fa, we note that z ^ can be represented as z = 7/(1 + 7) + it for some real 
parameter t G M^. We then have 

>(/(.)) ^ ^ , 0. 



r 



72 + (7 + l)2t2 1 + (7 + l)2t2 



Since t is decreasing as z travels along T^, we conclude that ^{f{z)) is decreasing. 

For z G F4 U F5, 2; can be represented as z = t + ni for some small constant n > 
and real parameter t. We note that for k = 0, 

^m-':^^>^. w.(o.i). 

Moreover 3?(/(2;)) is continuous for k around zero, uniformly for all t if t lies in a 
compact subinterval of (0, 1), which is indeed the case if 2; G F4 U F5. Hence we can 
choose K small enough so that 

sup ^{f{z))<^{f{z*)). 

zGr4ur5 



□ 



Now we are ready to find an asymptotic expression for the integral 



z^n(u) = [ e^M^^'<^-,) . ^Mim-m] . M . ^ — ^^^^^^ 



27r 9{z) [vMy^{z-p)f 

As in p], we first fix some 5 > small, then we decompose F = F' U F" where 

V := {zeV ■\z-p\<5}, F":=F\F'. (4.15) 

We can also define Fqo = F'^ U F'^ where F'^^ := z/M^/^(F' — p) is the image of F' under 
the map, and F'4 := FooVF'q^. We can also set 

n{u) = H'{u) + H"{u), n^{u) = H'^iu) + H'^iu). (4.16) 

Here Ti'^u) is the part of the integral fl4.14p on F'UF' and the definitions for 'H"{u),'H'^{u) 
and T-L'^{u) in (14.161) are similar. 

Our strategy to prove (14. 8 p in Proposition 13.11 is as follows. 

• By saddle point analysis, the integral of z in 'H{u),'Hoo{u) should be concentrated 
on the point z = p. Hence Ti'^u) and T-L'^{u) should be negligible, as is proved in 
Lemma 14.21 



For the integral of z near z = p,we prove in Lemma H]3] that the difference |'H'(m) 
H'Ml is small. 
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Now we discuss how we should choose S. First, by some very basic calculus we obtain 
that, if 5 < (p — l)/2, then for |z — p| < 5 we have 



:=C„. 



We then choose 6 > such that 



S < min 



p — 1 



6Co 2 

For such choice of 5, we have, for |-2 — p| < 6 



< max 



\s-p\<5 4! 



\z — p\ 



< Co\z — pI'^ < —\z — p\ 



(4.17) 



(4.18) 



(4.19) 



Therefore, for z G Fi n F', z = p + te^'"/^ for < t < 5. Hence (recall f"'{p) = -Iv^ in 



^(f(p + te'^-/^)-f(p))<- t'. 

o 



(4.20) 



Also, because of Lemma l4.ll we know that ^{f{z) — f{p)) for z G F" is even smaller. 
Hence we have 

mi^) - fip)) < -i^'Sye, Wz G F". (4.21) 

because of this, intuitively l-fiu) and "H'^ defined in fl4.16p are negligible, as the following 
lemma states. 

Lemma 4.2 If u is hounded below, i.e., u> U for some fixed U , then there exists some 
constant C, c > and Mq > such that for M > Mq we have 



\n"(u)\ < Ce-""e 



cu —cM 



\V!L{u)\<Ce-'^e 



cu„—cM 



(4.22) 



uniformly for u> U. 
Proof. We have, by flOTj) 



Zm'H"{u)\ < 



< 



7r(z/MV35)fc J^„ 



9{p) 



1 



-dz 



uMy^{z-p)f 
dz (4.23) 



We note that \g{z) \ is bounded above and below for z G F". Moreover since 
^{z -q) < -t/{2uM^/^), we have 



exp 



uM^l^u^{z-q) < Cexp 
< C exp 



vM^''\u-U)^{z~q) 
-e{u-U)/2\ =Cexp(-CM) 
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for some constants C and c. Hence from fl4.23p we have 

\Zm'H'\u)\ < CM-('=-^)/3g-c«g-.353M/6 ^ ^g-c«g-cM 

for some constant c,C > and for M large enough. The statement for Ti'^ (u) is proved 
in p. □ 

Now in order to prove (14.81) in Proposition 14. ![ we just need to prove 

\n'{u) - n'^{u)\ < Ce-'^'M-^l^ 

for some constant c, C > and for M large enough. First using change of variables 
a ^ vM^/^{z - p) in I^M) for Ti'^ (u), we know 



2n 



^uM'^/^u{z-q) _ ^-Mu^iz-pf/'i 



1 



[i/MV3(^_p)] 



-dz. 



(4.24) 



Let r = r'o U r; where := V n To and T\ := T n Ti. We can define V.'^ ^, H'^ ^ to be 
the integral in (I4.24p on Fq U Fq and F'^^ U F^ Similarly we can define I-L'q and l-i'-^. The 
following lemma completes the proof of the first part of Proposition 14.11 

Lemma 4.3 For some constant c, C > and for M large enough, we have 



Zm'H'q{u) — l-L'^ 



00,0 



.u 



< Ce-""M-l/^ I^Af^'i(M) - •H'^.i(m)| < Ce-'^'M-^''^ (4.25) 



Proof. Assume M to be large enough so that Fq = Fq- For z G Fq, h'M^/^\z — p\ = e/2. 
By (14191) we have that M{f{z) - f{p)) is bounded. Also 



|gA/(/(.)-/(p)) _ g-M.3(,_p)3/3| ^ CM\f{z) - f{p) - u\z - pfm 

< CM\z-p\^ = CM~^/^. 
Also because g{z) has no singularities or zeros around z = p, 



(4.26) 



< C\z-p\ = CM-^/\ 



Thus 



,M(/{^)-/{p)) 



gjp) 

9{z) 



< 



^M{f(z)~f(p)) _ ^-Mu^{z-pf/2, 



+ 



,M{f(z)-f{p)) 



gjp) 



1 



Since we have 



^ 



,M[/(^)-/(p)] 



'Mu^{z~pf /3 



|z/MV3(^-p)| 



(4.27) 
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the integrand in fl4.27p is bounded above by Ce ™. But the integral region is 0{M ^/^). 
Hence the first part is (14.251) is true. 

For the second part, we have z = p + te^'^'l'^ for e/{2vM^/'^) < t < 5. By 
^{f{z) - f{p)) < -uH^/Q. Like fICTD we have 



Also 



9{v) 



< C\z-p\ = Ct. 



From these two inequalities we obtain 



< Ce 



Hence, similar in fl4.27p 



POO 



e/(2i/Afl/3) 

POO 



dt 



l-k-l 



dt = Ce-^^'M-^'^ 



where in the second step we use the change of variables t v-^ t/ (2vM^/^). 



□ 



4.2 Analysis of J{v] 



This time we use the saddle point analysis for J{v) to prove (14. 9 p in Proposition 14.11 
Since most of the analysis will be similar to that of the previous part, we just outline the 
key steps. 

The contour of S in J{v) is graphed in Figure [31 Here S = ULo ^» Ui=o ^* where 
So = {a; = p+-^e-*:0<t<7r/3}. 



Si 

E2 



uj = p + te 



in/3 . 



3e 



2Z/M1/3 
V3 



<t<l 



UJ = p+-+it:t> 2 



As in the previous section, we choose S in such a way that the following lemma 
holds. 

Lemma 4.4 ^{f{u))) is monotonously increasing as u travels along Ei UE2. Moreover, 
for z = p+ l/2 + itET,2 (here t > \/3/2), we have 



^{f{co)-fico*))>C\o^ 



V3/2 



(4.28) 



for some constant C > 0, where u* is the interception o/Si and S2. 
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Figure 3: The Contour S in J{y). 



Proof. 



For w G El, w = p + te^'^Z^ gQj^g 3e(2z/M^/3)-i < t < 1. Then 



2p2 (^^2 ^ ^ p2 j + _ + - 1)2^ 

It is straight forward to verify that the right hand side is positive for t G [0, 1]. 
For a;GS2,a; = p + l/2 + zt. Then 

t{{2p - l)e + p{p - 1) + {2p - l)/4) c 



> - > 0. 

2 \ t 



p2 ^2 + + 1/2)2 j ^2 + _ 1/2) 

for some constant C > 0, uniformly for all t > \/3/2. Hence we have 

3?(/H-/K))>ciog' ^ 



73/2 



□ 



As in the last subsection, we can also define 5 to satisfy f l4.18p . Similarly we can also 
define 

i:' = {iueE:\tu-p\<5}, s" = s\s', 

and Soo = S'oo U S'^, where T,'^ := uM^^^iH' — p) is the image of S' under the map 
^ zyMi/3(u; -p), and S'^ = 

This time, for a; G Si fl S' we have 



sR(f(p + te^-/^)-f(p))>U' 

o 

and, because of Lemma holds, we have for cu G S", 



D 



(4.29) 



(4.30) 
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Specifically, for uj G S2, by Lemma [4.4[ 



z/3 



3fJ(/(a;) - /(p)) = 3fJ(/(a;) - f{w*)) + 3ft(/K) - f{p)) > -5^ + Clog(^-^J (4.31) 

for some constant C > 0. 

Similarly, from the decomposition of S = S' U S" and Sqo = S'^^ U E'^, we can also 
decompose J7'(f) and Joo{v) into 

Because of fl4.30p and fl4.3ip . the following lemma holds true. Since the proof is 
similar to that of Lemma we just provide the outlines. The only different part is that 
now our S is an infinite contour. 

Lemma 4.5 If v is hounded below, i.e., v > V for some fixed V , then there exists some 
constant C,c> and Mq > such that for M > Mq we have 

\ZllJ'\v)\ < Ce-^^e-^^^ mv)\ < Ce-^^e"^^ (4.32) 

uniformly for v > V . 

Proof. The result for J^{v) is proved in [T]. We just provide a brief proof about 
Z^l J" iv). The only difference from the previous subsection is that now S" is not of 
finite length. First since 3?(w — q) < e/(2z/M^/^), we obtain, similar to Lemma [4.21 that 



exp 



- uM^/\^{uj -q) < Cexp(-cw). 



for some C, c, uniformly for v > V. We decompose J"{v) into J"{y) = Ji{v) + J^liv) 
where J[{v), J^{v) are the integral of (jO]) on S" n Si and S" n Eg. Since S" n Si is of 
finite length, we can take the same procedure as in the previous subsection to prove that 



Now we consider J^ip)- For u G Sg, by using (14.3 ip we obtain 



\Zl^j"{y)\ < ^^MlH i g-<^Mi/3„5}(a;-g)g-M»{/H-/{^)) 



9{p) 



\vM'^/'\uj -p)\^duj 



00 



cM „—cu 



□ 



After showing that J"{v) is negligible, we then turn to analyze J'[v\ we can de- 
compose YI = Sq U S'l where Sq = S' fl Sq, S'l = S' fl Si. Also we can use the same rule 
to write S'^ = S'^ q U S'^^ 1. From this we can similarly decompose 

J'iv) = j^iv) + Jliv), JM = JLA^) + JL,iiv) 

As a analog of Lemma 14. 3[ we have the following lemma. The proof is similar and 
thus we just omit that. 
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Lemma 4.6 For some constant c,C > and for M large enough, we have 
uniformly for v bounded below. 



As a final remark, Lemma 14.51 and Lemma 14.61 together implies the second part of 
Proposition 14. 1[ 

4.3 Finishing the proof 

By fl3.8l) in Corollary 13.11 and by the scaling in fl3.13p we obtain 



P 



/ M=^/3(7-l)V3 



1\2 

1 I — Amir 



< X ^^det(/-/Coo) (4.34) 



7 

where /Coo is an integral operator on L^(0, 00) with kernel 

POO 

lCoc{u,v) = - Uooiy + X + u)Joo{y + X + v)dy. 
Jo 

for 'Hoo(w), Jooiy) defined in (14. 6 p and fl4.7p . By the same argument in section 3.3 of yj, 
we can prove that the Fredholm determinant in fl4.34p is just Fk{x). This finishes the 
proof. 



5 Proof of Part 2 of Theorem 11.1 



In this part, we analyze the limiting distribution of the smallest sample eigenvalue when 
there are some "true" eigenvalues being smaller than the critical point 1 — 7"^. 
We assume that for some 1 < < ^"2, 

We also assume that the rest of the 7r~^'s are in a compact subset of (vr^^, 00). In this 
case, our constants are 



1 1 7-2 / 1 1 e 

Here e > is a pre-fixed smaller number. We note here that due to some very simple 
algebra we can verify that z/ is a real number. Again, we define 

1 



^ ' l<r2 or N-ri+l<l<N-k 



n (^-^^)- (5-2) 



The function l-Liu), J{v) is now 



-Hiu) = / dze^'''''<^-'^~^ ■ e^^(^) ■ " . ^ (5.3) 

J{u) = '^^^ I due-''''''^'''^^-''^ ■e-^'f^'^^ ■ g{uj){uj-'KMfdu (5.4) 



22 



The constant Zm is defined to be 



Finally, like the previous section, we define our Tiociu) and J'oo{v) to be 



-HM = e-™Res„=o|e"'/'-^|, (5.6) 
Jooiv) = ^e'"" [ s'^e'" ds. (5.7) 

where 

T,oo = {z = 2e + it:t eR} 

oriented from bottom to top. We note that the integral on Sqo in (15. 7p also equals to the 
integral on the imaginary axis, from bottom to top. Again, in this section, our main goal 
is to prove the following proposition. 

Proposition 5.1 Assume u > U,v > V are bounded below, then 

1. There exists some constant C, c > 0, Mq > such that uniformly for u> U 

\ZMn{u) - noo{u)\ < Ce-^"M-^/2^ VM > Mo. (5.8) 

2. There exists some constant C,c> 0, AIq > such that uniformly for v > V 

\Z],^Jiv) - Jooiv)\ < Ce-^"M-i/2, VM > Mq. (5.9) 

In order to prove this proposition, we also need the saddle point analysis on f{z). 

Note that with the choice of /i in (15.11) . the two saddle points are z = hm and z = I/^iitin)- 
It is not hard to know 

1 < < < TTiV. (5.10) 

liTXN 7-1 

and 

/'(tt^v) = 0, f"{TiM) = -y". (5.11) 

In subsection 15.11 we use the residue theorem to provide an asymptotic analysis for 'H(m) 
in (15.81) . and in subsection 15. 2[ a saddle point analysis around ttat will provide us a good 
approximation for J[v) in fl5.9p . Finally, in subsection 15.31 we finish the proof. 

5.1 Analysis of l-i{u) 
By the residue theorem we have 

Zun{u) = UM + ^ / e^^^^^^-(^-) . e-^(/(^)-/(-)) ■ ^ ■ ^ dz 

(5.12) 
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where 

Hr{u) := ^e-™Res,=o|^ ■ e*^[^(--+'^/(^^))-^(--)l ^^""^^ ^ 1 (5.13) 

l« ^(vr^ + a/(z/VM)) J 

and r' is a contour inclosing tti, . . . , iiN-k but excluding 7rAr_fc+i = . . . = hn- We will later 
choose r' explicitly, based on which we will prove that the integral on T' is negligible. 
Hence the main contribution of f l5.12p is the residue 1-Li{u) in (I5.13p . 
The following lemma gives an approximation of 'Hi{u). 

Lemma 5.1 For u > U hounded below, uniformly there exists constants c, C > and 
Mo > such that for all M > Mq 

\Hi{u)-V,oo{u)\ < Ce-'^'M-^I^ 

Proof. The proof is almost the same as that in [1]. 

First by expanding /(ttat — a/{v\fM)) — f{'nN) and on g{T(N) / g{T^N + a/{v\/M)) around 
a = we obtain 

g{TrN + a/{iyy/M)) 

where 0m (a) is a analytic function around a = 0. Moreover all the coefficient in the 
Taylor expansion of 4>m{o,) around a = are bounded in M. Hence we obtain 

n{u) = n^u) + -^Qj,j{u). (5.14) 

Here Qm{u) is a polynomial of u of degree at most fc — 1, and all the coefficients olQuiu) 
are bounded in M. Due to the factor e~'^^ in fl5.14p we complete the proof. □ 

To keep our promise our next step is to prove that the second term in fl5.12p is 
negligible. This is done in Lemma (5.31 But before that, we need to choose our contour 
r' explicitly. Again, our contour F' is symmetric with respect to the real axis thus we 
just define the upper part. We take F := IjLi ULi where the Fj's will be defined 
below. For a graph of the contour please refer to Figure |H 

• Take vr' such that max(7rjv_fc, 7/(7 — 1)) < vr' < ttat. Moreover take some k > to 
be a sufficiently small constant. We take Fi to be the quarter-circle centered at vr', 
with radius k. Here k > is a small constant. 

• Take R = 1/ ^JJi — 1. We define 

Fa = {2 = i: + m : 1 + i? < t < vr'}. 

• Define F3 to be the arc centered at 1 with radius i? := 1 / ^JJi—1. Using some simple 
algebra, we can verify that with our choice of the real part of the intersection of 
F2 and F3 is between l/(/i7r7v) and 7/(7 — 1), provided that n is small enough. 
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Figure 4: The Contours T and E 



• If 1 - i? < 7/(7 + 1), we just define 14 = {7/(1 + 7) + it : t G M} to be the vertical 
fine joining and T^. This is shown in Figure ID If 1 — i? > 7/(7 + 1), then we 

just exclude r4 in the definition of F' = ULi ^ ULi connecting F3 and F5 
directly as we did in the previous section. 

• Take vr* = min(7ri, 7/(7 + 1)), we define 

T5 = {z = t + iK : TV* < t < max(l - R, 7/(1 + 7))}. 

• We finally define Fg to be the quarter-circle centered at tt* with radius k. 
Our analysis of 'H(u) replies heavily on the following lemma. 

Lemma 5.2 There exists some constant c > such that 

sup 3ft f{z) - /(ttat) < -c. 
zer' L J 

Proof. Similar to Lemma 14. ![ we also divide the proof into several parts. 

• In F2, z can be represented as z = t + in. We note that when k = 0, 
z = t ^ {1/ fiiiN, vttv) and 

dt ^ " t{t- 1) 

Thus 3ft(/(2;)) is strictly decreasing for t traveling on the real axis ttjv to l/(/i7rjv). 
Moreover ^{f{t + in)) is continuous for k, around zero, uniformly for all t lying in 
a compact subset of (l/(/x7rjv), 7i"Ar)- Thus we can choose k to be sufficiently small 
such that for some constant c > 0, 



sup 3ft f{z) - /(ttat) 



< -c. 
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In Fs, z = 1 + i?e*^ for in a subset of (0, it). Then 



A. 



RsinO 



< 



l + 2/?COS^ + i?2 

i?sin^ 



+ + RcosO) - 1 







l + 2i?COS^ + i?2 

with our choice of R := l/^/JI — 1. Hence 3fJ(/(z)) is decreasing in T^. 

In r4(if it exists), ^ = 7/(1 + 7) + it. With exactly the same calculation as that of 
Lemma [4. H we can prove that ^{f{z)) is decreasing in 

in Fs, z can be represented as z = t + in, just as that in This time when k = 0, 
z = t e (7r*,7/(l + 7)) and 

dt ^ " ^(1 -i) 

With the same argument as that in we can choose k sufficiently small such that 

sup 3? f{z) - /(tin) < -c. 
^ersurs L 



□ 



Based on Lemma [521 our next lemma establishes that the integral along F' in fl5.12[) 
is negligible, which finishes the proof of (15. 8 p in Proposition 15.11 

Lemma 5.3 There exists some C,c > and Mq > such that uniformly for u > U 
and for M > Mq 



27r ir' 9{z) [vM^^iz - txm)Y 



dz 



Proof. For 2; G F', by Lemma [5.21 we have 



,M5R(/(^)-/(7riv)) 



< e 



-cM 



Moreover, since uM^^'^'St^z — g) < — e for z E F', by the same proof as that in Lemma [4.21 
we have e^'^^ '^u^{z-q) ^ Ce^'^^ for some constant c, C > 0, uniformly for u > U and for 
M sufficiently large. One last notes that 



1 



< C 



g{z) [uM^/^{z-n^)] 
remains bounded for z G F'. This finishes the proof of Lemma [5.31 



□ 
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5.2 Analysis of J{v) 

The contour for J{v) is fairly simple compared to F. As is also shown in Figure HJ the 
contour S is defined by 



U = TlM + 



2e 



z/Mi/2 



+ it:te 



Now if we have a constant 6 > such that < 6 < {tin — 1)/2 then for u such that 
la; — ttatI < 5 we have 



:=Co. 



We then fix 6 to satisfy 

Then for — 7Tn\ < 6, 
f{oo) - /(vr^) 



(5.15) 



(5.16) 



r(vr^) 



< max 



\ns)\ 



|s-7r]v|<<5 3! 



w — vttvI 

2 



(5.17) 



Now we decompose S = S' U S" where 

S' = {w G S : |u; — vttvI < 6} 

and S" = We further define S'^ := pM^''^{T.' - ti^) to be the image of S' under 

the mapping. Moreover define S'^, := Soo\S'o^. Based on the integration contour, we can 
also set 

J{v) = J'{v) + J"{v), J^v) = JM + JM (5.18) 

where the decomposition is similar to that of the previous subsection. 

Define a;* as the intersection of S' and S". From fl5.17p and from the parametrization 
cj* = TTiv + 2eM-^/'^/v + it* for some t* bounded below we obtain 



f{uj*) - fin^) 



> 



3fJ(a;* - vrjv) -—\bj*-'Kj^ 



+ -vH*^ > —6^ 

M A - 32 



(5.19) 



if M is large enough such that t* > 6/2 and 3e'^/M < 

To analyze the behavior of J{v), we have the following lemma. 



Lemma 5.4 9ft(/(ci;)) is increasing as u travels along S. Moreover, foru G S" and for 
sufficiently large M , we have 

?R(/(u;) - /(vTA,)) > Ci + C\og{t/5) (5.20) 

for some constant Ci, C > and t is defined by oj = tt^ + 2e/ (z/M^/^) + it. 
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Proof. The proof is straightforward as we observe for to = tin + 2e/ (z/M^/^) + zt, 



It 



(72-1)^2 + ^(^-1)2-^2 



72(^2 + t2)((^_ 1)2 + ^2) 



(5.21) 



where A := t^n + 2e/(z/M^/2)_ ^^^g ^-j^g^^ ^2(^^ _ 1)2 _ ^2 > q -g equivalent to that 
^ > 7/(7 — 1), which is indeed true. Hence fl5.2ip is positive. 
For the second statement, if t > 5, then from fl5.2ip 

for some constant C. Integrating this from t* to t, tonether with fl5.19p gives the desired 
result. □ 

For /(cj), = tttv is the saddle point. So we expect that all the integral on S" and 
S'4 is negligible, as the following lemma states. 

Lemma 5.5 If v is hounded below, i.e., v > V for some fixed V , then there exists some 
constant C,c> and Mq > such that for M > Mq, v > V, 



Z-,,'J"{v)\<Ce-''^e 



cv -cM 



|J"(^^)l<C7e-™e 



cv „—cM 



Proof. The result for J^{v) is proved in pLj thus we just prove the result for Zl^J"{v). 
Recall that 



TT 



Mi/2( 



1 k 



u - tin) 



Note that by Lemma [5.41 we have 



-M5R(/M-/{7rjv)) 



< e 



where t is the paralyzation of w = tttv + 2eM ^^"^ / v + it. Moreover e = e . 

Finally g{uS){uj — t^^Y is just of polynomial growth of t. We get that 



ZllJ"[u)\ < Ce 





t 


Is 


6 



-cM 



\tn^/^dt < Ce'^^'e- 



for some constants C,c> and for M sufficiently large. 



□ 



As expected, our next step is to prove that Zj^j J'{v) and J^{v) are sufficiently close 
to each other. This is proved in Lemma 15.61 

Lemma 5.6 If v is hounded below, i.e., v > V for some fixed V , then there exists some 
constant C,c> and Mq > such that for M > Mq, v > V, 

\ZM'J\v)-J'^{v)\<Ce-^^M-'l\ 
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Proof. By the change of variables s i— )■ uM^^'^(co — ttat) in J'^ in (15 .711 we have 



\Z7,'j'(v)-J'(v)\ < 



TT 



du 



uM^/^{u -ttn) . (5.22) 



Now we follow the same proof as that in Lemma [4.31 thus we just list the key steps. We 
can prove 

where s is defined by w = TT AT + (z/M^/^)^-'^s. Further \g(uj)/g{TTN) — l\ < CM~^/'^s. Thus, 
by the change of variables uj = tin + {h'M^/'^)~^s in (I5.22p we obtain 

-eii p2e+ooi 
^ J2e-ooi 

for some constant c, C > 0. □ 
5.3 Finishing the proof 

Using the same steps as in pLj, with minor modifications, we can prove 

where Hek{x) is the k-th probabilists' Hermite polynomial, that is, 

He,{x) :=(-l)V^/2(^Aye-^V2. 

Thus the kernel ]C{u,v) has the desired form, as is proved by [T]. We omit the steps. 

6 Independence of the extreme eigenvalues 

In this section we will prove that the largest eigenvalue and the smallest eigenvalue are 
asymptotically independent. In Section |4] and Section |5] we dropped the subscript one 
for notational convenience. In this section, however, we will consider both the maximum 
and the minimum eigenvalue. Hence we need these subscripts to distinguish them. 
Note that we have four cases here. 

• Case 1. £i = vrf ^ < 1 + 7"^ and ^at = vr^^ > 1 - 

• Case 2. = vrf ^ > 1 + and In = t^n^ > I - 7""^- 

• Case 3. ii = vrf ^ < 1 + 7"^ and ijsf = n'^^ < 1 - 7"^ 

• Case 4. ii = n^'^ > 1 + 7"^ and = tt^^ < 1 - 7^^ 
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Since the proofs for all the cases are almost exactly the same, thus here we just prove the 
theorem for case 1. 

Recall that in Proposition 13.1^ we proved that 

P(6 < An.in < A^ax < 6) = dct (l-( f}' f'^]]. (6.1) 



K2I K22 



where the operators Kji are defined in (13.31) . Our first observation is that for any constant 
Wm, we will have 

< A„,„ < A„„ < « ^ de.(/-(^ £)(o wli,)) 

We will leave the constant Wm as an normalization constant, to be defined later. 

As the next step, to ensure the probability to be non-trivial, we need to take the 
proper scaling of and ^2- Under the case 1, define 

6 = Ml — T^x, G = P'2 + TT—y- (6.3) 

Here ^i, Vi and ai are defined in f l4.ip . The constant parameter /i2, 1^2 and 02 are defined 
in yy by 

/i2:= 1 + - , ^2:= ^ , «2 = ^. (6.4) 

V 7/ 7 3 

Our next step is analyze Kji. Let's take Kn as an example. As we did in (13.131 - 
I3.18p . consider the kernel ii'ii(?7,C), with the change of variables 

we know that K^^ ■ ^^(0,^i) ^^(0,6) is equivalent to /Cn : L'^{0, fiiW^ /ui - x) ^ 
L^(0, /iiM"^/z/i — x) defined in (13.141) . The analysis for the other Kji^s are the same. 
Finally we can arrive at the following equation. 

n^rain < X, A^ax < V) = det (^I - J^^'^^^ ] ] (6.5) 



Here 



7M2/3 r _ 1 ~ 7M2/3 



:i-7-') -An.i 



_ 1)4/3 ' ^ '"™°J ' ''^^'^ ■ + 1)4/3 



A. 



Amax - (1 + 7"')' 



is the proper scaling of Amin, Amax and the )Cji{i,j G {1, 2}) is the integral operator from 
Ji to Jj where Ji := (0, /iiM^^/z/i — x) and J2 = (0, 00). The kernels on the diagonal are 

POO 

/Cii(M,f) = - / 'Hi{y + X + u)Ji{y + X + v)dy (6.6) 
Jo 

/C22(M,f) = / H2{y + X + u)J2{y + X + v)dy (6.7) 
Jo 
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with l-ii{u) , Ji{v) defined by (13.151) and (13.161) . The 'H2{u) , J^2{v) appeared in (16. 6p are 
defined in [I] by 

n.iu) := "^^^ f e-''^''''^'<'-'^^'^e-''^^'-'-'>'h'"\\^—dz (6.8) 

A/fi/3 r ^ 
:= ^^^^ / T\{yj _ vr,)da;. (6.9) 

In Section m we have aheady shown that 'Hi(m), Jli{v) will tend to the desired limit at the 
speed of M~^/^ as M tends to infinity. In [T], Baik, Ben Arous and Peche showed that 
'H2{u) , J^2{v) will also tend to the desired form at the speed of M~^/^. That is to say, 
the diagonal part in (16.51) has the non-trivial limit because of the correct scaling in (16.31) . 
In order to and Amax are asymptotically independent, our strategy is to 

show that the off-diagonal terms Wm^2i and W^lCi2 in (16. 5p . properly scaled, will tend 
to zero. Hence the matrix in (16. 5p will tend to a diagonal matrix. Since the determinant 
of a diagonal matrix equals to the product of the determinant of its diagonal parts, the 
joint probability P(Amin < a;, Amax < y) is approximately P(Amin < a;)P(Amax < y)- Now 
we carry on our strategy in details. 

First consider the off-diagonal terms in (16. 5p . First define the constant 

•^•^^ eMf2{p,) ■ ^2(^2) ' (z/2MV3)te- ^''•'^^ 

(pi,P2 and the functions /i(-)) /«>(■) 5 ?2(') "will be defined below). 
Using the change of variables as we did in (I3.13l - l3.18p . we get 

W^j^}Ci2iu,v) = - ^ ^^^3 J J + ^)"^2(i/ + -i^; 1^) ■ — ^ — , (6.11) 



WMK.2iiu,v) = TJTn [ dz [ duH2iy + u;z)Ji{x + v;u) 



u — z 



Here 



92iP2) hMV3(2_p2)]fc^' 

Mv;co) = e-*^(^^(-)-^^(P^))e-^i"^^'''"(--''^)-|^-[z/iMi/3(u;-j9i)]^-^ 

The function /i(z), /2(-2), ^i(z), ^2(-2) are defined by 

fi{z) = -fii{z - qi) + log z - ^\og{z - 1), ie{l,2}. 

N-ki r2 ^ 

g,{z) := {z-ir n 92{z):={z-ir \{ —— 

^■^ Z — TXf Z — TXf 

£=Ar-ri+l £=fc2+l 
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6.12) 

6.13) 
6.14) 
6.15) 
6.16) 

6.17) 
6.18) 



and the parameter pi,p2,qi, q2 are defined by 

7 7 



Pi 



P2 



qi=Pi + 



q2 =P2 



(6.19) 



From some simple calculation, we know that P11P2 are the saddle point of fi{z) and 
2), respectively. Below we will perform our saddle point analysis on W^^K,i2 and 
WmK,2i- Let's start from W^/]Ci2{u,v) first. In Figure Owe plot the contour Fi and S2 
in (16.111) . The contour Fi is defined in Figure [2], Section HI Moreover the contour E2 is 
defined in p. We quote their definition below. 




Figure 5: The Contour Fi and S2. 

Define E2 := ULo ^2,fc U |JLo ^2,fc where 

52.0 := {P2 + 3e{u2M^/^)e"^^-'^ ■0<9< n/3}, 

52.1 := {P2 + te^'^'^ ■■ 3e(z/2Mi/3) < t < 2^2}, 

52.2 := {p2 + 2p2e^'^/^ -x:0<x<R}, 

52.3 := {-R + i{V3p2-y):0<y<V3p2} 

where is a pre-fixed constant which is sufficiently large and independent of M. 

The intuition of the proof is as follows. The saddle point of fi{z),f2{z) is pi,p2 
respectively. Moreover in our contour we leave the saddle points along the steepest 
decent direction, in (16.111) the integral on z will be mostly concentrated on the point p2 
and the integral on oj will be mostly concentrated on the point pi. But by our choice 
of the constant Wm, Hi{x + u; z) and J2(|/ + v]uj) will be bounded at these two saddle 
points. Moreover around these two points {uj — z)^^ ~ {jp2 — Pi)^^ will also be bounded. 
Hence due to the factor M~^/^ in front of the integral of (16.111) . W^llCi2{u,v) will tend 
to zero, at the speed of M~^/^. To paraphrase our intuition into a rigorous proof, we will 
arrive at the following lemma. 

Lemma 6.1 These exist some constant C > 0, c > 0, Mq > such that uniformly for 
u > U,v > V , we have 

\Wm^}Cu{u,v)\ < C{e-'^-''')M-^/\ (6.20) 
\Wm1C2i{u,v)\ < C(e-^"-^'')M-^/l (q21) 
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Proof. Here we just prove (16.201) as the proof for fl6.2ip will be similar. Let's define 
6 < min{pi/10,p2/10, (pi — P2)/10} to a sufficiently small constant independent of M. 
Just like what we did in Section H] and we decompose the contour by Fi := T[ + T'(, 
S2 := S'2 + E'2' by 



r[ = {zeri \z-pi\<5}, 



S2 = T.2\T.2. 



We first analyze Fi. By Lemma [4.11 we know that 3fJ(/(z)) is decreasing on Hence 
there exist some constant c such that 



yz G F;, ^ f{z) - f{p,) < 0, G F'/, 3ft f{z) - f{p 



< -c. 



For S2 we can also deduce the same result. From Lemma 3.2 in [1], we have 



Vw G K 



21 ■ 



/(^) - /(Pi) 



> 0, 



Vw G S'2', : 



> c. 



(6.22) 



(6.23) 



For z G F']^, we note three results proved before. (1) expluiM^^^u^^z — qi)] < exp(—eu/2) 
by the definition of the contour; (2) \'gi{z) /'gi{pi)\ is bounded; and (3) \uiM^^^(z — 
< e"''^ is bounded. These three results, together with (I6.22p give that 



/ \Hi{u; z)\dz < Ce- 



(6.24) 



for some constant C,c> 0. Now we proceed to prove 



\J2{v; u)\dijj < Ce 



Similar to (14. 181 - 14. 2ip . we can choose 6 small enough such that for \u —p2\ < S, 



(6.25) 



3ft 



6 



Actually in P the authors proved that (I6.26P holds as long as 



S < min 



2(1 + 7) '6(43 + 4(1 + 7)^ 



Now for a; G 



92{P2) 



92[UJ) 



^2 



Ml/3 



lk2 



UJ-P2\ 



(6.26) 



(6.27) 



since that |^2(P2)/?2(i^)| is bounded and that e-'^^x^^ is bounded for x > 0. From (I6.27P 
we obtain (I6.25P directly. Having analyzed the behavior of Hi, J2 on F'^^ and S2, our next 
step is to prove that they decay exponentially fast on T" and S2. Let's take Hi{u; z) as 
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an example. The proof of J2{v]u) will be similar. Indeed, by the second part of fl6.22p 
and by the fact that dii^) /di{Pi) is bounded, we obtain 



Hiiu; z)\dz < Ce-^'^e-^^'. (6.28) 



Similar results for J2(f ; oj) can also be obtain. 

As the last step in proving our lemma, we have 



Here 



iy^i/Ci2(M, v) = -^jj^ [h + I2 + I3 + h\ . (6.29) 



Ji := dz dujHi{x + U] z)J2{y + V]u) ■ , (6.30) 



UJ — z 



and J2, /a, Ia, are just the integral of fl6.30p on (F'/, Sg), (F'^, Sg) and (F", S2), respectively. 

In Ji, z G T\ and uj G S2, we have |a; — z\~^ < ((pi — p2)/2)^"'^ is bounded above. 
Hence by fl6.24p and fl6.25p we have 

< Ce~"^-™. (6.31) 

For /2, z G F'/,a; G Sg. We just use the trivial bound |cj — < CM^^^ for some 
constant C, by the definition of our contour. Hence by f l6.28p and fl6.27p we get 

I/2I < c'e-^"-"''e-'=^^ ■ < Ce-'"-"''e-'=^. (6.32) 

With similar analysis, we can prove that fl6.32p also holds for J3 and J4. Note that in 
f l6.29p we have an extra M~^/^ in front of the integral. This finishes the proof of fl6.20p 
in our lemma. □ 



As a simple consequence. Theorem 11.21 holds. Here's the rest of the proof for the 
theorem. 

Proof of Theorem li.M By Lemma (16. ip we have 

||VrA//C2i||i < CM-^/^ \\Wm1Ci2\\i < CM-^/^ (6.33) 

for some constant C. Here || ■ ||i is the trace norm of the operator. Since the Fredholm 
determinant is a locally Lipchitz continuous function with respect to the trace norm, from 
(16. 5 p we obtain 

P(Ami„ < X, A^ax < y) = detf/-f./" 

= det(/-/Cn)det(/-/C22) + C(M-i/3) 
-> Fk,{x)FkM- 

□ 
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7 Conclusion 



In this paper, we studied the spiked population model to establish two results: (1) the 
asymptotic distribution of the smallest eigenvalue of the sample covariance matrix, and 
(2) that the largest and the smallest eigenvalues are independent. Our approach is based 
on the convergence of operators under the trace norm. 

For the smallest eigenvalue, when the spike is weak (all above the threshold 1 — 7"^), 
the local fluctuation is of order (9(M~^/^), and will converge to the generalized Tracy- 
Widom law under the correct scaling. When the spike is strong enough to pull the smallest 
eigenvalue out of the Marcenko-Pastur sea, it wil fluctuate at the order of C(M~^/^), 
following the generalized Gaussian distribution. We also proved that the largest and 
smallest eigenvalues are independent. Combined with the asymptotic behavior of Amax 
proposed in [T], we can establish the joint distribution of (Amin, Amax)- 
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